Multimodal Propositions? From Semiotic to Semantic Considerations in the Case of Gestural Deictics

We call utterances that comprise elements that are perceived by different sense modalities and are coded according to a non-linguistic code multimodal utterances (cf. (Fricke, 2012)). An example for non-linguistically coded signs are indexicals like pointing gestures, which, according to Peirce, bear some nomological, causal or attentional relation to their objects (CP 1.372, 2.248, 2.285 1) In situated dialogue, interlocutor’s frequently use multimodal utterances like definite descriptions plus pointing gestures in an exophoric way. Accordingly, such deictic acts are a starting point for looking for multimodal propositions: Whereas the linguistic expressions are treated as arbitrary symbols that are interpreted intensionally with respect to a world or a circumstance according to some standard model theory, the nonlinguistic element, if it indeed follows a nonlinguistic code, must, by definition, interpreted in a different way. At least three distinctions have been made at first, however (Levinson, 2008):